1. Field of the Invention
The present invention relates to the field of interconnects for computer systems and in particular to ordering interconnect transactions in a multi-node computer system.
2. Description of the Related Art
Many industry standard and proprietary interconnects require transactions on the interconnect to be ordered according to certain rules to maintain memory consistency throughout the system, to enable proper functioning of cache coherency protocols, and to avoid livelock, deadlock, and other undesirable conditions. The PCI-X Extensions to the Peripheral Component Interconnect (PCI) specification allows for the general ordering rules to be relaxed for certain transactions, by specifying a “relaxed-ordering” attribute.
In a modern computer system, there is a great advantage to be gained by using the PCI-X relaxed ordering attribute. The PCI-X relaxed ordering attribute indicates that a PCI-X initiator can allow the data written to reach memory in any order. The attribute can be used for the “payload” part of an I/O transfer, which usually makes up the bulk of the I/O transfer from an I/O adapter. The performance gain can be attributed to at least three factors: First, by using the relaxed ordering attribute, cycles from one adapter are not blocked behind cycles from another adapter. Second, if multiple memory controllers exist in the computer system, write data can be sent to each memory controller simultaneously rather than waiting for each write to receive a completion notification before running the next. In a multiple memory controller system, memory can be “striped” across the memory controllers, so even a contiguous block of write data can target several separate memory controllers. Third, because each memory controller is typically an independent entity, the writes are processed much faster than if a single memory controller was processing the entire stream of writes.
Simply allowing relaxed order writes to be run whenever they occur can result in problems, however, because of the interaction of relaxed order write cycles with non-relaxed order (regular) writes. As the PCI-X specification outlines, relaxed order writes may pass regular order writes, but not vice-versa. Thus, relaxed order writes are free to be run when they are received by core logic comprising an I/O subsystem for a computer system. Regular order writes, however, must wait until ordering of all previously accepted writes is guaranteed before they can be run. The result is that under some circumstances a stream of relaxed order writes can starve one or more regular order writes, degrading system performance and potentially causing failures due to timeouts within the operating software. If the core logic simply runs relaxed order writes as soon as it receives them, it may starve out other regular order writes.
Transaction ordering queues (TOQs) are known for enforcing the ordering rules of interconnect protocols, ensuring that interconnect transactions will execute in an order consistent with the ordering rules. As such, not all interconnect transactions typically go into TOQs, but only those transactions for which ordering rules apply. Transaction ordering queues are typically implemented as first-in-first-out (FIFO) queues.